Global Range Encoding for Efficient Partition Elimination

Skipping mechanisms have been extensively studied to improve query performance over large data volumes. A powerful skipping technique for in-memory columnar databases is partition elimination. The goal is to eliminate, as much as possible, loading physically partitioned data into memory and probing column partitions against queries. This is achieved by consulting column partition summaries. The summary is often very compact compared to the column partition itself, and is kept in memory, e.g. the MinMax zone map. These summaries have been extensively integrated into modern in-memory database systems including SAP HANA [6]. In this paper, we argue that probing byMinMax range is not efficient when there are gaps in the values that appear in a column partition. Any predicate that needs to probe values in a gap inside a MinMax range naturally ends up requiring a candidate check; this reduces the benefits of column partition pruning. To address this problem, we propose a mechanism to encode each partition (likewise, query) using global ranges, carefully designed to reduce false positive rates. Our approach not only provides a compact in-memory representation, but also supports efficient partition pruning using bitwise operations. Compared toMinMax, our experiments support that our approach significantly reduces the false positive rate. It can allocate memory budget among ranges in partition groups, based on column density, estimated false positive rates from recent workload, and gaps.